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ABSTRACT: Technologies to communicate construction project information (engineering designs, schedules) 
have evolved into a wider range of innovative ecosystems for engineering practices (e.g., cloud-based 3D 
representations and advanced immersive environments). There is a lack of exploration of effective user interaction 
for learning and training in relation to how presented information influences cognition in these ecosystems. The 
presented research investigates the users’ cognitive and attentional differences using the interactive capabilities 
of Mixed reality (MX) technology. The enhanced user-situation interactions are analyzed by measuring cognitive 
dynamics with an emphasis on two processes (attentional focus and cognitive load) in relation to the challenge of 
the engineering learning task— defined by its complexity (limited time frame for observations of the situations, 
number of required observations) and nature (episodic). Cognitive dynamics were measured using an 
electroencephalography (EEG) device that senses electrical activity in response to changing levels of cognitive 
stimuli via electrodes placed on the scalp. Measuring fluctuations in cognitive processing (related to the intensity 
of various task demands) allows associating efforts on semantic information processing for learning and training 
tasks (e.g., walkthroughs for safety checks in job site in MX). The approach enhances opportunities to design 
technology that best adapts to the user needs for engineering practices with an efficient comprehensive 
performance assessment. 
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1. INTRODUCTION 


Construction sites are characterized by their dynamic nature, as they are filled with a multitude of activities and 
potential risks. Safety in construction is a critical aspect of production activities and a major priority effort for 
successful implementations in construction organizations (Guo et al., 2017). Construction safety training is of the 
highest priority across the industry, and the use of technology intervention has facilitated such efforts (Frank Moore 
& Gheisari, 2019). The provision of construction safety training plays a pivotal role in cultivating a safety-oriented 
environment within the construction sector. Ensuring optimal safety in the construction industry necessitates a 
collaborative endeavor involving various stakeholders, including owners, designers, construction companies, 
workers, regulators, and educators (Sacks et al., 2013). Typically, prior to commencing work on a construction site, 
workers are mandated to complete an Occupational Safety and Health Administration (OSHA) 10-hour 
construction training program. This program is delivered online and encompasses safety-oriented lectures, videos, 
and slides. 


The efficacy and significance of this training program, as well as its adequacy, are continually pertinent inquiries 
(Wilkins, 2011). There have been efforts focused on implementing a more effective construction safety training 
program using different methods like personalized training programs or training with virtual reality devices 
(Jeelani et al., 2020). However, VR technology implementations generate potential risks for the user. For example, 
they don’t easily enable representing at-scale safety requirements in the VR environments for the users’ own 
exploration in training. Another possibility of risk is that VR applied to OSHA safety training may become a new 
source of distractions to users (Asish et al., 2022), impacting the intended outcome of training. 


Despite the widespread use of technologies in training (including VR and AR as interventions), methods that reveal 
the effectiveness of the technologies as training approaches are not incorporated into the training programs. For 
example, methodologies for assessments employ paper-based exams or supervised self-reports—which have 
considerable limitations— to determine subjects’ performance before and after the training program (Jeelani et al., 
2020). It is critical, therefore, to comprehensively assess the effectiveness of inventory training tasks with the use 
of technology by considering the individual characteristics of the trainee (technology user) as they factor or are 
subrogate into the overall performance. There is a need to find alternative methods to assess the efficacy and 
benefits of implementing safety training interventions due to individual differences and self-report methods' 
disadvantages—ranging from response bias, recall bias, and subjectivity to cultural and language barriers. The 
researchers anticipate that incorporating the users’ individual performance front and center might facilitate a 
smooth path to successful training programs. 
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The presented work uses Mixed Reality (MX) technologies. MX combines real and virtual worlds for the creation 
of environments where users can function and interact in the physical and virtual worlds. MX has facilitated the 
consolidation and analysis of activities in the physical space, such as in production processes involved in the 
manufacturing of goods or services—.e., the activities for converting raw materials into finished products. The 
co-existing of real and virtual interaction allows connections and reactions between virtual objects and the physical 
space, undoubtedly facilitating the enhancement of the study of construction activities, including training programs 
in the industry. For example, by “moving” the construction site production activities into just a small physical 
space for training. 


This study presents a novel assessment method utilizing electroencephalography (EEG) technology. The EEG is 
used to measure electrical activity in the brain, which output data brings insights into the timing and nature 
(rhythms in brain activity across frequency bands— delta, theta, alpha, beta, and gamma) of the underlying 
cognitive processes. The presented research proposes the utilization of the EEG technique for safety training and 
problem-solving tasks. The method enables the automatic collection of data from individual trainees to study the 
effectiveness of training tasks. The approach collects the neural response when learners attempt to address 
challenges in training tasks under conditions of complexity, such as the cognitive effort involved in solving a 
question. By conducting an analysis of the EEG data, this technique provides information to assess the training or 
problem-solving tasks through cognitive load and attention degree analysis for individuals, providing information 
on which task the user performs well or has deficits. The outcome can easily correlate to individual differences 
(cognitive abilities such as memory, attention, perception, and problem-solving skills) to find the effectiveness of 
the overall training tasks. For example, what is the impact of attention deficits on particular training tasks? 


The presented study focuses on the problem-solving process in order to develop a more effective, precise, and 
unbiased approach to assessing performance in training. 


2. BACKGROUND 


Since the early 20th century, scientific studies using electroencephalography techniques have experienced a 
considerable evolution. Mainly these efforts involve the detection and analysis of minuscule electrical signals 
emanating from the human brain during its various activities (Sanei & Chambers, 2007). Electroencephalogram 
(EEG) signals are categorized into distinct power bands corresponding to various brain wave frequencies, 
facilitating the identification of different states or conditions of ongoing brain activity in humans (see Table 1) 
(Fernandez Rojas et al., 2020; Klimesch, 1999; Zietsch et al., 2007). 


Table 1: Frequency and statements of brain waves. 


Brain Waves Frequency (Hz) Statement 
Delta 0.5-4 Idling and sleep 
Theta 4-8 Mental fatigue and mental workload 
Alpha 8-13 Mental workload, cognitive fatigue, and attention or alertness 
Beta 13-30 Visual attention, short-term memory, and working memory 


The literature has established the meaning of spectral powers of various EEG waves and cortical locations in 
evaluating cognitive load during problem-solving tasks. Researchers observed an increase in the power of both 
theta and alpha bands as task difficulty escalated, suggesting a direct association between these bands and cognitive 
load (Sarailoo et al., 2022). More specifically, the augmentation of theta spectral power serves as an indicator not 
only of heightened task complexity but also of enhanced working memory capacity (Borghini et al., 2012). 
Additionally, the beta band can potentially serve as another indicator of cognitive load and working memory during 
tasks. In visual working memory tasks, there has been an observed augmentation in beta activity within the parieto- 
occipital channels (Mapelli & Ozkurt, 2019). 


Building on early definitions of attention from William James, in his book the Principle of Psychology, James 
states that attention “is the taking possession by the mind, in clear, and vivid form, of one out of what seems several 
simultaneously possible objects or trains of thought.”(James, 1890). As a condition of selective awareness, 
attention degree controls the quality of one's task-solving. Enhancing one's ability to regulate attention pertains to 
the domain of executive attention, also known as controlled attention. This cognitive process encompasses 
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functions such as planning, decision-making, and problem-solving (Fernandez-Duque et al., 2000). Executive 
attention refers to the cognitive ability to deliberately redirect one's focus from one task to another or to inhibit the 
processing of extraneous information. This study focuses on the focus degree, as known as the intensive of 
attention, as one of the layers of information that help with analyzing performance. To indicate the state of attention 
degree, delta, theta, and alpha waves are the most used (Kaushik et al., 2022). 


Multiple studies have documented an elevation in mid-frontal theta activity, a reduction in central and parietal 
delta activity, and a decrease in frontal and parietal alpha power during states of attention (Kaushik et al., 2022). 
Additionally, the relative magnitudes of spectral power across various waveforms can serve as an indicator of 
attention levels. As per the findings of the researchers, the attention ratio, referred to as the theta/beta ratio, 
possesses significant utility as an indicator for the analysis of attention. Moreover, it has been demonstrated by 
researchers that a robust association exists between attentional degree and the ratios of theta/beta, theta/alpha, and 
alpha/beta (Derbali & Frasson, 2011; Ghasemy et al., 2019; Hillard et al., 2013). More specifically, a larger ratio 
of alpha/beta indicates a more concentrated situation, in the meantime, there is a negative correlation between the 
ratio of theta/beta and the focus degree (Derbali & Frasson, 2011). 


While EEG is commonly associated with medical and neuroscience applications, it also has some interesting and 
potential applications in the field of construction. Applications like worker cognitive load or stress level monitoring 
could help to improve construction on-site workers’ health, well-being, and productivity (Jebelli et al., 2018; Saedi 
et al., 2022). The productivity of construction workers is not solely determined by their individual workload but is 
also greatly impacted by their emotional state, particularly when encountering hazardous work conditions or 
confined spaces. The utilization of wearable EEG headsets for monitoring the emotional state of on-site 
construction workers is a potential avenue for construction managers to enhance control and optimize the overall 
workflow of building projects (Hwang et al., 2018). 


Given that construction workers consistently operate under conditions of high stress and heavy workloads, the 
matter of safety is a critical domain that researchers seek to enhance. The studies on the EEG in the construction 
site may lead to the optimization of the construction safety programs. For example, assessing the on-site worker’s 
mental workload via EEG could help managers identify individuals who are not in their best mental status and 
better arrange human resources to reduce risk and hazards (Chen et al., 2016). On the other hand, ensuring that 
personnel remain focused on their hazardous tasks and that they are not easily distracted by external factors is 
consistently crucial. Wearable EEG devices promise to identify factors of distractions of construction workers in 
hazardous tasks and to improve construction site safety (Ke et al., 2021). 


In addition to calls for its application in on-site construction workers, EEG has been utilized in laboratory studies 
based on virtual reality (VR) to enhance the performance of building or construction environments. The utilization 
of virtual environments offers a valuable opportunity to replicate real-world scenarios. By incorporating EEG band 
power scalp mapping into machine learning models, it becomes possible to assess the authentic responses of 
individuals residing in a building space. This analysis can encompass several aspects, such as comfort, pathfinding, 
and spatial utilization (Zou & Ergan, 2023). Beyond the analysis of the fatigue level from EEG signals collected 
in the virtual environment, collected EEG data can help the development of new models to improve the prediction 
and prevention of construction fall hazards (Tehrani et al., 2022). 


3. METHODOLOGY 


The presented approach is a model for the performance and assessment of individual trainees’ problem-solving 
tasks implemented in an MX environment combined with an EEG headset. The MX environment will provide a 
virtual simulation of the training tasks, and the EEG headset will collect EEG signals for cognitive analysis on the 
problem-solving task. This research work employs the Theta (4-8Hz), Alpha (8-13Hz), and Beta (13-30Hz) 
frequency bands primarily to assess the cognitive load and attention levels exhibited during a problem-solving 
task. 


Subsequently, the provision of performance feedback entails the comprehensive processing of all collected data. 
The flow of the feedback methodology process is presented in Fig.l. Detailed steps will be discussed in the 
following sections. 
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Taking a OSHA Taking the 
Sign up for the Start the constrction OSHA 
experiment experiment related pre- construction 
———— exam safety training 


— 


Subject Pre-experiment testing and 
PEE related OSHA knowledge 
training 


Taking the walk- 


through construction Data collection Data Comparison 
OSHA safety checking processing and conclusion 


exam with MR device 


Problem-solving Task-related data Data synchronize 
task collection and processing 


Fig.l: Workflow of the problem-solving task performance feedback process. 


3.1 Subjects’ recruitment and OSHA safety training 


The experiments involve recruiting a sample of fifty individuals aged between 18 and 35 years old, who are 
enrolled in a university program with a background in civil and construction engineering. The experimental 
procedure will be conducted within a laboratory area measuring 5 meters by 5 meters. 


Upon enrollment in the experiment, participants are required to respond to a pre-test that will inform the knowledge 
of OSHA construction safety training. After the pre-test, subjects are required to watch the OSHA construction 
safety training video and then take the OSHA construction safety examination in a virtual environment after 
becoming familiar with the manipulation of the MX device. 


The OSHA construction safety training video refers to the selected OSHA construction regulations (Huang et al., 
2003; "Top 10 Most Frequently Cited Standards | Occupational Safety and Health Administration,"). Fig. 2 is an 
example of the applied OSHA construction standard. 


1926.451(e)(1) 
When scaffold platforms are more than 2 feet (0.6 m) above or below 
a point of access, portable ladders, hook-on ladders, attachable 
ladders, stair towers (scaffold stairways/towers), stairway-type 
ladders (such as ladder stands), ramps, walkways, integral 
prefabricated scaffold access, or direct access from another scaffold, 
structure, personnel hoist, or similar surface shall be used. 
Crossbraces shall not be used as a means of access. 


Fig.2: Example of an OSHA Construction Safety standard applied in this study. 


The current approach classified the selected construction OSHA standards for violation identification in the MX 
environment scene into three tiers. The tiers are designed based on the complexity of violation identification, 
meaning the user’s level of effort required—i.e., the complexity is related to the steps to determine the violation 
in a virtual scene. Table 2 presents the details of complexity tiers for violation of OSHA standard identification. 
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SECTION A - EXTENDED REALITY TECHNOLOGIES IN CONSTRUCTION 


Table 2: Tiers and level of effort on OSHA standard violation identification in the virtual scene. 


Tiers Level of effort based on complexity of tasks for violation identification 
Tier 1 Direct visual contact with the objects. 
Tier 2 Need to search for information to infer a violation. 
Tier 3 Need to perform actions in to determine violations 


Tier 1 is applied to those violations that can be perceived through direct observation in scenes within the MX 
environment—1.e., there is minimum learner’s effort to perceive the visualizations (visual representations) that 
produce the stimulus for the learners’ identification of the violations. Within this tier, most of the violations could 
be identified by just a single observation of the virtual objects (visual representations). Typical examples for this 
tier are conditions that represent personal protection equipment through virtual objects (i.e., visual representation 
of the worker without properly using protection equipment). Tier 2 is for conditions in the scene that demand the 
user to search for information to infer the violations. The learners’ search for information is possible by triggering 
actions in the MX environment. The learner’s action for information search serves as an additional mediation 
mechanism for inference to determine violation (e.g., the learners’ virtual rotation of an extinguisher in the MX 
scene to determine the expiration date). Tier 3 complexity consists of additional actions for inference using virtual 
instruments to determine violations. Using instruments for inference implies an additional level of complexity, as 
other cognitive capabilities (spatial and reasoning abilities) are required to determine violation (e.g., displacing 
instruments to measure distances in the virtual space). An example of Tier 3 is the learner’s required action of 
using an instrument to calculate the distance that informs whether it’s a violation or not (e.g., the placement of a 
straight ladder against a wall). 


3.2 Problem-solving task design in the MX environment 


The design consists of creating an examination of on-site construction OSHA safety checks. The examination is 
framed as a problem-solving task to draw boundaries of complexity in the search space— the possible 
configurations, number parameters, and elements in the MX environment that impact the users’ decisions and 
courses of action. 


Users (learners, trainees) are required to review the compliance of safety standards of construction of small 
commercial buildings (3 stories distributed over 40,000 Sq ft) as a project engineer from a local subcontractor 
company. While wearing the MX device (Hololens 2), users are asked to inspect and label safety standards 
violations in the first story of the building within 10 minutes. Fig. 3(a) shows the MX ongoing construction site 
prototype. Fig. 3(b) shows a scene where the user is required to use an additional instrument within the MX 
environment to determine a violation, like the measuring tape within the problem-solving task. 


yd 


G Varta construction site (b) Integrated virtual commands for labeling virtual 
: objects. 
Fig. 3. Virtual environment for OSHA training. 


In order to facilitate unrestricted exploration of the entire construction site within a virtual environment, a 
navigation system was devised to enable virtual movements in the virtual world based on physical displacement 
in the real world (laboratory space). The navigation system allows unconstraint virtual displacements in the MX 
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environment. Due to the constraints on mobility in the physical environment, individuals are required to physically 
traverse the laboratory area in order to investigate and examine the construction site comprehensively. Instead of 
taking the examination totally virtually like a video game, this paper is trying to configure a balance between 
virtual and reality through this navigation system. 


3.3 Data collection 


The experiment design includes the collection of video, audio, and EEG data from a MX device and an EEG 
headset (see Fig.4(a) and 4(b)). The EEG data is collected from an OpenBCI Mark IV headset with a sampling 
rate of 125 Hz. The OpenBCI Mark IV headset includes 16 channels (with additional reference and ground 
electrodes in Al and A2) placed on the subject’s scalp according to the international 10-20 system. This paper 
collects EEG signals by using all 16 channels (FP1, FP2, F3, F4, F7, F8, C3, C4, T7, T8, P3, P4, P7, P8, Ol, and 
O2) as presented in Figs. 5(a), 5(b), and 5(c) (Homan et al., 1987; "Ultracortex Mark IV | OpenBCI 
Documentation,"). 


(a) Mixed-reality and EEG device. (b) Subject wearing the combined headset. 


Fig. 4: MX with an EEG device integration. 
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(a) Used channels in 10-20 system. (b) Channels numbering. (c) EEG heat map while example 
subject ongoing data collection. 


Fig. 5: Map of electrodes used in this paper’s EEG data collection process ("Ultracortex Mark IV | OpenBCI 
Documentation,"), and heat map example for EEG signal processing. 


During the implementation of the OSHA construction safety inspection, the researchers collect video, audio, and 
information on tagged violations by the subjects. All tagged violations will include the real timestamp information, 
which could help with synchronizing time stamps across different data streams and activities in the MX 
environment. 
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SECTION A - EXTENDED REALITY TECHNOLOGIES IN CONSTRUCTION 


3.4 Post-processing 


A synchronization task follows. It consists of mapping the generated streams of data collected during the 
experiment with the real-time stamps associated with each technology device. The timestamps were used to 
identify the exact occurrence of violations during the experiments. Once violation occurrences are identified, the 
EEG data will be segmented into 20-second epochs for target sections (20 seconds before each incident violation 
was labeled or tagged). The researchers post-process EEG data in MATLAB (version 2023a) and the EEGLab 
toolbox (Delorme & Makeig, 2004). After importing raw EEG data and electrodes’ locations correspondingly, a 
FIR filter is applied to bandpass filter the EEG data to a frequency of 0.5-30 Hz to help to remove low-frequency 
drifts and power line noise. Artifacts from eye blinks and muscle movement were corrected by applying ICA in 
EEGLab (Winkler et al., 2014). 


The next steps are the extraction of features of theta (4-8Hz), alpha (8—13Hz), and beta (13—30Hz) to analyze and 
compare the mental cognitive load and attention degree across different levels of efforts. 


4. EXAMPLE OF DATA RESULTS AND DISCUSSION 


The current experimentations of the treatment and control groups are in progress. The following is an example of 
the typical experiment and data captured for one subject, including the data processing outcomes for the treatment 
group. 


The presented example shows an experiment with a subject of the treatment group who has never taken any OSHA 
construction safety-related training. The researchers asked the subject to take a personalized OSHA safety training 
session. Once the training session was finalized, the researchers asked the subject to be immersed in an MX reality 
ecosystem by wearing the MX and EEG devices. The immersive environment consists of a virtual construction 
site with multiple scenes and situations that present safety violations and hazardous conditions based on OSHA 
standards. Each violation fell into three different tiers of complexity. As an illustration, the violations’ type and 
complexity tier are presented in Fig. 6. 


Violation Action to determine Violation Action to determine ¥ Action to determine 
category the violation category violation (step 1) violation (step 2) 
Personal 
protection 
equipment (PPE) 
Each scaffold shall 
Y be capable of 
supporting its own 
The hard hat should Visual contact with the virtual weight and at least 4 Awareness for information 
be wom all the time object to times the maximum Visually contact with the virtual search and completion of 
on the job site. check and determine the intended load applied object simple calculation 
violation. toit. 
(a) Easy-level complexity violations. (b) Mid-level complexity violations. 


Violation Action to determine W Action to determine Y Action to determine 
category violation (step 1) violation (step 2) violation (step 3) 


Conduct a free-body 
analysis of the 
suspension scaffolding 
and calculate the 
intended force on each 
suspension rope. 


Each suspension 


rope shall be capable 
of supporting at least 
6 times the 
maximum intended 
load transmitted to 
that rope. 


Reasoning task: free-body 
construction analysis 
of virtual objects. 


Visual contact with the virtual Awareness for information 
object. search task. 


(c) Hard-level complexity violations. 


Fig. 6. Example of complexity tier violations used in the experiment. 
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The easy-level complexity violations (like PPE violations) would only require subjects to visually be in contact 
with the virtual object to check and determine the violation. For mid-level complexity violations (like mobile 
scaffolding capacity violation), the subject needs first to be in contact with the virtual object (as a target virtual 
object with potential violation)—next, the subject searches for information to make inferences and verify a 
violation. For hard-level complexity violations (like suspension scaffolding capacity violations), the subject 
requires not only the awareness of an information search task to deduce a violation but also be involved in a 
reasoning task. The reasoning task demands constructs a free-body analysis of virtual objects. 


Fig. 7 (a) shows the result of cognitive load when the subject was experiencing different efforts associated with 
the tiers of complexity violations. As introduced in the previous sections, to analyze the cognitive load, the relative 
band power of Alpha and Theta was computed (Sarailoo et al., 2022). Based on the result, it can be concluded that 
to address the increasing complexity of tasks, individuals are required to exert a higher cognitive load to arrive at 
a solution. Furthermore, since the subject's cognitive load increased with the higher complexity levels when 
solving a task, it is possible to determine the dynamics of success and failure of each subject (cognitive dynamics 
of each subject). The dynamic enables the research to correlate the efficiency of the technology for training and 
individual differences in training tasks. 


Cognitive load Attention (selected channels) 
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(f) Attention degree (ratio between theta and 
beta). 


Fig. 7. Data from cognitive processes from one subject involved in MX training task. 


To analyze the attention degree when subjects address solutions in problem-solving, the band power of mid-frontal 
theta, central and parietal delta, and frontal and parietal alpha were computed. Besides, the ratio between theta and 
beta was also included as an indicator (Derbali & Frasson, 2011; Kaushik et al., 2022). As presented in Fig. 7 (b) 
(c), with the increase of the task complexity, the related power band of the mid-frontal theta increases, and the 
central and parietal delta decreases, which indicates a better intense of attention was put into the problem-solving 
task. However, the relative band power of frontal and parietal alpha exhibited a little increase as the task complexity 
escalated, indicating a potential decline in attention levels during the problem-solving activity. Besides, the ratio 
between theta and beta was also not presented as the ideal model. This result may be caused by lost calibration or 
bad connections between some channels of EEG collecting headset and subject’s scalp. Drawing precise reasons 
on this issue is challenging due to the insufficient number of trials and subjects involved. 
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Nevertheless, by analyzing the cognitive load condition and the attention degree while the subjects were facing 
the problem-solving task, the researchers could infer that some mistakes made by the subject were not because of 
the inefficiency of the training program but due to loss of attention or the lack of ability to keep on a high cognitive 
load level for a long time. In this example, the subject stayed focused during the whole session and solved all three 
violations (presented in Fig.6) successfully. However, the presented example contains a short number of decisions 
with only simple construction scenes and a limited number of violations. It’s relatively easy to keep focused on 
the problem-solving task. For subjects who face a more complex scene and can’t solve all violations, the reason 
for the mistakes (i.e., performance on correct inferences to solve the problem) will become part of the analysis, 
including the effectiveness of the MX intervention for the training program. 


5. CONCLUSION AND FUTURE WORK 


The presented research is a successful design and development of a method for the effectiveness of assessment 
safety training. The approach includes in its development the design and construction documents of the 
construction project site to build a virtual construction site. The information was used to build an MX environment 
for the learner’s self-exploration using a navigation system, enabling the learner to mimic the real workplace with 
the advantage of a mixed-reality device. The method uses EEG signals to estimate cognitive conditions that inform 
the users’ effort in decision-making while solving a problem relating to OSHA violation (i.e., virtual safety 
inspections in the MX environment). The technology consists of an MX device and a 16-channel EEG headset. 
Subjects could walk freely in the experimental space, as the portable EEG and mixed-reality devices allow them 
to collect the data wirelessly to a local network set for the experimentation. With the model developed, the 
researchers could successfully and accurately assess the subject’s cognitive load and attention levels while solving 
the construction safety-check problem. The outcome provides new and comprehensive information that helps to 
analyze the performance during learning and problem-solving. With the developed method, the researchers could 
overcome the bias from self-report evaluation or any paper-based test and get a comprehensive and personalized 
performance analysis. For future work, the researchers will model the effects of the cognitive load and attention 
degree analysis by applying machine learning algorithms for inference on the subjects’ behavior during problem- 
solving tasks. 
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